Live Orchestral Piano, a system for real-time orchestral music generation

This paper introduces the first system performing automatic orchestration from a real-time piano input. We cast this problem as a case of projective orchestration, where the goal is to learn the underlying regularities existing between piano scores and their orchestrations by well-known composers, in order to later perform this task automatically on novel piano inputs. To that end, we investigate a class of statistical inference models based on the Restricted Boltzmann Machine (RBM). We introduce an evaluation framework specific to the projective orchestral generation task that provides a quantitative analysis of different models. We also show that the frame-level accuracy currently used by most music prediction and generation system is highly biased towards models that simply repeat their last input. As prediction and creation are two widely different endeavors, we discuss other potential biases in evaluating temporal generative models through prediction tasks and their impact on a creative system. Finally, we provide an implementation of the proposed models called Live Orchestral Piano (LOP), which allows for anyone to play the orchestra in real-time by simply playing on a MIDI keyboard. To evaluate the quality of the system, orchestrations generated by the different models we investigated can be found on a companion website 1 .